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Executive Summary 


The Quality Assurance Report (QAR) Program, which includes assessment and coaching, is a 
continuous improvement process to support early childhood programs. In 2022, Missouri 
Department of Elementary and Secondary Education Office of Childhood (DESE OOC) 
contracted with the Early Childhood Innovation Center (ECIC) at the University of Missouri- 
Kansas City-Institute for Human Development (UMKC-IHD) to implement the QAR Pilot Program 
for a sixth cohort of 250 classrooms in 50 early childhood programs. The project was expanded 
in 2023 by enrolling new programs (some from Cohorts 1-5) for a total of 500 classrooms. An 
additional 250 classrooms were added in Fall 2023. For all classrooms, DESE OOC is using the 
Classroom Assessment Scoring System®(CLASS). This report describes the first two years of 
implementation of the program. 


Since February 2022, ECIC completed the launch and installation of the project. This included 
assigning or hiring highly qualified staff members, onboarding new staff, training staff in the 
assessment instruments, and training staff in the coaching model. This year, ECIC incorporated 
CLASS affiliate trainers to the QAR team. This action directly supports state-wide capacity- 
building for QAR implementation. Early childhood quality specialists (ECQS) are located 
throughout the state to ensure comprehensive program implementation. 


Installation also included contacting and enrolling early childhood programs to confirm their 
participation in the cohort. To date, 88 programs representing 698 classrooms are enrolled. 
Slight changes to the initial classroom assessment protocols include shifting from the CLASS 
Pre-K assessment instrument to the Second Edition of the CLASS Pre-K assessment instrument 
and adding the CLASS Infant/Toddler and CLASS Pre-K Environments tool. Missouri is on the 
forefront of implementation with these instruments since they were published within the past 
year. Preliminary CLASS data are available for 548 classrooms (89 infant classrooms, 172 
toddler classrooms, and 287 Pre-K classrooms). Overall, scores are slightly higher than national 
norms. 


Coaching sessions occurred on a regular basis and reports were analyzed. A total of 4,643 
coaching visits with educators were completed in 2023. Coaching sessions mostly occurred in- 
person (74.5%). The majority of coaching interactions focused on building partnerships (44.7%), 
which is the foundation of Practice-Based Coaching. For 34.2% of visits, the focus was on shared 
goals and action planning. Educators’ feedback indicate they are reflecting on their practices 
(4.49), improving on their practices (4.35), and deciding their own goals (4.37). Educators 
reported less efficacy in making progress on their goals (3.84); however, many of them have 
only recently engaged in coaching, so it is expected that goal attainment ratings will improve 
over time. 


Results suggest that using CLASS for the QAR program is a promising approach. It appears that 
the design of the program and the coaching model are scalable, feasible, and well-received by 
program administrators. Baseline data collected and analyzed in this work will be used to 
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examine the effects of data-informed coaching. This study can be used to inform future efforts 
regarding quality improvements in early childhood programs in Missouri. 


Background 


In December 2019, DESE received a Preschool Development Grant Birth to Five (PDG B-5) grant 
from the U.S. Administration for Children and Families, U.S. Department of Health and Human 
Services (DHHS) and the U.S. Department of Education (DOE). This grant provides funding to 
coordinate a “more effective, high-quality early learning system that prepares children to be 
ready for school.” One activity for the PDG B-5 grant is to pilot the Quality Assurance Report 
(QAR), a continuous improvement process for early learning programs and to provide families 
with information about quality programs. The Pilot Program is authorized by the Missouri 
Legislature. The QAR employs evidence-based practices with a goal to ensure that all children 
have access to high-quality early childhood care and education (ECCE). The project is designed 
to intentionally include children with disabilities, special health care needs, and those in rural 
areas with limited access to early childhood programs. 


Since 2019, five cohorts of early childhood programs participated in the QAR Pilot Project. Their 
participation included evidence-based classroom environment assessments and on-site 
consultation from early childhood professionals. The DESE OOC was interested in exploring a 
different model for the sixth cohort, using the Classroom Assessment Scoring 

System ® (CLASS). CLASS is an observational assessment tool and a recognized quality 
improvement system that measures teacher-child interactions. DESE OOC is also interested in 
testing the efficacy of a model that provides assessment and technical support related to 
program administration, before providing supports in classrooms. 


Project Design 


In 2022, DESE OOC contracted with the ECIC to implement the QAR Program for Cohort #6 and 
evaluate its effectiveness. The work plan included assessing approximately 250 classrooms in 50 
programs. In 2023, DESE OOC expanded the contract with the ECIC to implement the QAR 
project with all currently participating programs and additional newly enrolled programs. This 
work plan involves serving a total of 500 classrooms. 


Programs new to the project were encouraged to attend a virtual orientation webinar. These 
webinars were offered at multiple days and times. Webinars outlined timelines, project design, 
and coaching implementation. After the webinar, the programs were contacted to begin the 
coaching process. This includes a packet of information that outlines the coaching agreement, 
information about the assessment tools, and information about coaching implementation. The 
first visit with the teacher included reviewing the coaching agreement and coaching model, a 
focused process of getting to know the teacher, and defining preferred coaching strategies. The 
second visit with the teacher included information regarding assessments, as well as initial goal 
planning. 


ECIC conducted a comparative study to examine the effectiveness of two models: 


Model # 1: Classrooms are assessed using the CLASS instrument and teachers engage in 
coaching for continuous quality improvement (CQI) focusing on effective teaching practices 
over a period of 16 months. 


Model #2: In addition to classroom assessments using the CLASS, programs in Model 2 are 
also assessed regarding administrative policies and practices using three instruments: the 
Program Administration Scale’ (PAS), the Organizational Equity Assessment’ (OEA), and 
the Early Childhood Work Environment Survey’™' (ECWES). Administrators engaged in 
coaching and technical support for CQI at the program level during 2022, focusing on 
effective administrative practices. In 2023, ECIC re-assessed programs regarding 
administrative policies and practices to measure improvement. If programs showed 
progress and met an established program quality threshold, they received CQI coaching at 
the classroom level, focusing on effective teaching practices for the remainder of 2023. If 
programs did not meet the quality threshold, they continued to receive coaching on 
administrative practices for the duration of the project. 


For the comparative study, ECIC is using a pre-/post- research design to measure improvement 
over time, with classroom interactions (CLASS scores) as the primary outcome variable. 
Programs were randomly assigned to either Model #1 or Model #2. All classrooms, regardless 
of their model assignment, received initial CLASS assessments to establish a baseline 
measurement and a second assessment (at Time 2) using the CLASS at the end of the study. The 
comparative analysis will show change-over-time for the two models. A final report will be 
prepared in summer 2024 to describe the process and outcomes of the study. 


ECIC is using the Practice-Based Coaching“ (PBC) model for structured coaching 
implementation. PBC is a cyclical process for supporting teaching and administrative staff’s use 
of effective practices that lead to positive outcomes for children and programs. Fundamental to 
PBC is Establishing the Partnership, an essential element to coaching cycle components that 
ensures collaboration between the coach and coachee. The coaching cycle components are: 1) 
Shared goals and action planning; 2) Focused observations; and 3) Reflection and feedback. 
ECIC is using a hybrid-coaching format, with monthly on-site coaching sessions, and monthly 
virtual coaching visits with teachers and administrators. 


Classroom interactions using the CLASS continue to be used, along with an environmental 
assessment using the CLASS Environments. PBC model for structured coaching interventions 
continues to be the method for interactions between ECQS and educators. 


Process Evaluation 
ECIC Staffing 


ECIC assigned multiple staff in a variety of roles to work on the QAR Project. Staff roles include 
principal investigator, project coordinator, data analyst, team leads, ECQS, and logistical 
support staff. 


ECQS are responsible for assessing and coaching programs. A total of 24 positions were 
available, with 21 of those positions filled by the end of 2023. Applicants were recruited from 
local, regional, and national networks including the Regional Hub network, the Missouri 
Association for the Education of Young Children, Missouri Head Start Association, the certified 
CLASS assessor registry, the American Educational Research Association, National Head Start 
Association, and the National Association for the Education of Young Children. These 
organizations were encouraged to forward the recruitment posting throughout the state and 
across the country. Specific requirements for the ECQS positions included: 


e Bachelor’s degree or higher, 

e Experience in early childhood education, 

e Experience working with adults (specifically training, coaching, or mentoring), and 

e Willingness to engage in field-based, remote work (including travel within the state). 


The QAR staff represent a highly qualified array of educational backgrounds and work 
experiences. Of the five staff that are in leadership positions with the project, two have 
doctoral degrees, one has an educational specialist degree, and two have master’s degrees. Of 
the 21 ECQS, eleven (55%) have master’s degrees and ten (45%) have bachelor’s degrees. They 
also bring a broad range of experiences to the project: school district administration, early 
intervention, classroom-based instruction, early learning program management, music therapy, 
and non-profit administration. 


Staff Onboarding. In addition to standard UMKC and ECIC on-boarding processes, staff for the 
ECQS positions participate in a series of on-going staff development activities. 


e Classroom Assessment Scoring System (CLASS) Reliability Training — CLASS training is 
completed through synchronous, on-line training sessions for staff offered by 
Teachstonet? initially and then by ECIC affiliate trainers. Staff then complete rigorous 
reliability testing through Teachstone for each of the three instruments: Infant, Toddler, 
and Pre-K. 


1 Teachstone, Inc., is the owner of the CLASS instrument and exclusively provides products and training for 
observer certification, on-going support, and training for the affiliate model. (https://teachstone.com/products- 


and-trainings/ ) 


e Practice-Based Coaching - Coaches complete an online, asynchronous training on 
Practice-Based Coaching (PBC)*, with a two-day follow-up training, either in-person or 
virtually. 


Observation Tool Training and Reliability. To maintain fidelity with program implementation, 
ECIC requires ECQS to be trained and achieve reliability in the CLASS instruments to conduct 
observations. All of the ECQS were trained and achieved reliability in CLASS as reported in Table 
1. They are also required to be trained in PBC before they enter the field for coaching duties. 


Table 1. Observation Tool Reliability 


Number of Staff 


Observation Tool Trained Percent Reliable 
CLASS — Infant ali 26 | 100% 

_ CLASS — Toddler | 26 | 100% 
CLASS — Pre-K (2"¢ Edition) | 27 | 100% 
CLASS — Infant/Toddler Environments | 26 | N/A?3 
CLASS — Pre-K Environments | 23 | N/A? 
CLASS — Pre-K Affiliate Trainer 7 | 100% 
CLASS — Infant/Toddler Affiliate Trainer | 4 100% 


On-Going Program Staff Development 


To ensure the remote program staff continue in their professional growth and to ensure 
consistency in program implementation, a variety of meetings are scheduled with staff on a 
regular basis. 


e Team Meeting — The QAR project team gathers monthly to discuss program 
implementation. Topics have included human resources reminders, scheduling/training 
updates, coordination of service delivery, and other administrative topics. An agenda is 
provided, and agenda items are solicited prior to the staff meeting. 


e Professional Learning Community (PLC) — Project team members meet twice monthly 
to engage in a robust book study. Team members rotate leading the PLC meeting. Team 
members completed the study of Conscious Discipline: Building Resilient Classrooms“ in 
March 2023. Project team members were queried in February 2023 to select a topic for 
the next book study. Of the topics generated, ECQS requested further study in culturally 
competent coaching practices. The book Hearing All Voices: Culturally Responsive 


2 The Pyramid Model Consortium offers online training in the Practice-Based Coaching model 
(https://www.pyramidmodel.org/ ). The ECIC Leadership Team offers additional training the model for application 
in the QAR Program. 

3 The CLASS — Infant/Toddler Environments and CLASS — Pre-K Environments were released in 2022 and tools for 
establishing reliability are not yet developed. The ECIC is participating in the reliability study for these instruments. 
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Coaching in Early Childhood” was selected and study began in May and continues 
through 2023. 


e Coaching Roundtable — ECQS meet monthly to attend this open-format meeting where 
they bring questions and ideas centering around implementation of PBC with educators. 
This meeting is facilitated by the Team Leads. 


e Conference Attendance — Selected QAR program staff attended and presented at 
conferences to contribute to their professional development and to represent the QAR 
Project to the greater early childhood community. Staff attended Conference on the 
Young Years, Teachstone InterAct, the National Association for the Education of Young 
Children Professional Learning Institute, the BUILD Conference, and the National 
Association for the Education of Young Children Annual Conference. 


QAR Information Dissemination 


Several activities for information dissemination about the QAR Program have been available 
throughout the first half of program implementation. 


e Conference on the Young Years (CYY) — The QAR Project Coordinator presented two 
sessions at CYY in March 2023: What is the Missouri Quality Assurance Report? and All 
About CLASS. Each session was attended by approximately 35 educators. Also, ECIC 
hosted an informational booth at CYY. Approximately 600 informational fliers about the 
QAR project were distributed and 30 programs submitted applications as a result of this 
recruitment activity. 


e MO-SECA Leadership Academy — QAR project fliers were included in participant packets 
for the Missouri Supporting Early Childhood Administrators (MO-SECA) Leadership 
Academy in March 2023. Approximately 219 administrators attended this academy. 


e InterAct Conference — The first in-person conference hosted by Teachstone was held in 
April 2023 and attended by approximately 350 individuals from across the U.S., 
including the ECIC QAR project team. The QAR Project Coordinator presented a session: 
Missouri Quality Assurance Report: Using CLASS to Improve Teaching Practices. Also, the 
ECIC QAR leadership met with Bridget Hamre (author of the CLASS Instruments) and 
with Teachstone’s research and evaluation team to share about developments in the 
QAR and explore opportunities for collaboration. 


e InterAct Now! — The virtual companion to the InterAct conference, InterAct Now! was 
held the first week of May 2023. The Project Coordinator presented two sessions at the 
conference: Missouri Quality Assurance Report: Using CLASS to Improve Teaching 
Practices and Using Innovative Methods to Analyze CLASS Data for Targeted Coaching 
Interventions. 


Program Enrollment 


There are 199 early learning programs with 698 classrooms enrolled in QAR. Of these, 107 
classrooms continued from Cohorts 1-5, 200 classrooms are enrolled from Cohort 6, 189 
classrooms were recruited and enrolled in Spring of 2023, and 202 classrooms were recruited 
and enrolled in Fall of 2023. Fall 2023 classrooms include Missouri Foundations for School 
Success and Missouri Quality Pre-K grantees. Table 2 reports information about the enrolled 
programs. 


Table 2. Information about QAR Enrolled Programs (n=199) 


# % 
Number of Programs 199 
Number of Classrooms 689 
Program Size 
Family Child Care 5 2.5% 
Small (1-5 classrooms) 143 71.9% 
Large (6+ classrooms) 51 25.6% 
Region (Classrooms) 
North Central 28 4.1% 
Northwest 9 1.3% 
Northeast 17 2.5% 
Kansas City Area 84 12.2% 
Central 108 15.7% 
St. Louis Area 233 33.8% 
Southwest 146 21.2% 
Southeast 64 9.3% 
Average Child Enrollment Capacity 63 
# of Programs Receiving State Subsidy fl 55.8% 
# of Program Receiving Head Start Funding 23 11.6% 


Outcome Evaluation 


Classroom Assessments 


CLASS Overview. The CLASS is an observational tool used to evaluate the process quality of 
early childhood classrooms. The CLASS focuses on observed teacher-child interactions through 
a series of four short observation rounds (15 — 20 minutes per round depending on the 
instrument used). After each observation round, each dimension is scored on a scale of 1 to 7. 
Scores are determined by scoring indicators for each dimension low, mid, or high. CLASS 
instruments are tailored to different age groupings: infant (0-18 months), toddlers (19-35 
months), and preschool (3 — 5 years). Beginning in 2022, Teachstone published a revised and 
updated version of the Pre-K CLASS tool (2 Edition). The QAR Project used this instrument to 


assess preschool classrooms beginning in 2023, along with the CLASS Infant and CLASS Toddler 
instruments. Table 3 outlines the domains and dimensions for each CLASS instrument. 


Table 3. Domains and Dimensions of the CLASS 


Pre-K — 2" Edition 
(3-5 years old) 


Responsive Caregiving Emotional and Behavioral Emotional Support 
e Relational Climate Support e Positive Climate 
e Teacher Sensitivity e Positive Climate e Negative Climate 
e Facilitated e Negative Climate e Educator Sensitivity 
Exploration e Teacher Sensitivity e Regard for Child 
e@ 


e Early Language Regard for Child Perspective 
Support Perspectives 
e Behavior Guidance 


Engaged Support for Classroom Organization 
Learning e Behavior 
e Facilitation of Management 
Learning and e Productivity 
Development e Instructional Learning 
e Quality of Feedback Formats 
e Language Modeling Instructional Support 
e Concept 


Development 
e Quality of Feedback 
e Language Modeling 


CLASS Results. A total of 548 CLASS observations were completed through November 30, 2023. 
Of these, 89 Infant classrooms, 172 Toddler classrooms, and 287 Pre-K classrooms were 
assessed. There are 150 observations remaining that will be observed through December 2023, 
as well as some programs that are not opening until January 2024. Preliminary CLASS data show 
that, overall, scores in Missouri programs tend to be higher than the current national averages. 


Scores are grouped by domains and use a seven-point scale. Scores of 1 to 2 are considered 
ineffective, 3 to 5 indicate a mix of effective and non-effective interactions, and 6 to 7 indicate 
effective interactions are consistent. There is one domain for infants, two domains for toddlers, 
and three domains for Pre-K children. 


The average infant domain score for Responsive Caregiving was 4.65. The toddler score for 
Emotional and Behavioral Support was 5.32 and 3.18 for Engaged Support for Learning. For 
Pre-K, Emotional Support averaged 5.71, Classroom Organization averaged 5.15, and 
Instructional Support averaged 2.89. 
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The relational climate/positive climate and teacher sensitivity dimensions are the highest 
scoring items in the mid-high range, with scores of ranging from 5.44 — 5.88. Dimensions 
regarding implementing concept-related content to children and language support were in the 
low to low-mid range, with scores ranging from 2.59 — 3.50. These initial results are consistent 
with reported national averages. Domain and dimension average scores are reported in Table 4. 


Table 4. Class Instrument Preliminary Average Scores 


CLASS Instrument Average Score 


sponsive Caregiving 


beteseactg sialic | 
Ce 
| PositiveClimate | SAT 99 | 5038 
aaa 


3.43 


CLASS Pre-K(n=287) | 
a al 
fF Positiveclimate——C«|SSBB| |S 
0.39 
[TeacherSensitivty ————s«dT~—=S«sSAO | ~~ 
| Regardfor Student Perspective (| 483 ~—~| 0.99 ~+| 436 
Classroom Organization SS SS a a aaa 
a eS el 
326 | oo4 | 285 —* 


11 


CLASS Environment. Beginning in 2023, new programs also had the CLASS Environment 
assessments conducted in their classrooms. The CLASS Environment measure is a supplemental 
tool designed for use in conjunction with the CLASS observation. They describe and measure 
features of the environment that can support effective interactions. The CLASS Environment 
assessments are divided into two instruments: Infant-Toddler (birth — 35 months) and Pre-K - K 
(36 months — Kindergarten). The Infant-Toddler tool consists of 32 items (14 in Emotional and 
Behavioral Support and 18 in Engaged Support for Learning) and the Pre-K-K tool consists of 29 
items (10 in Emotional Support, 8 in Classroom Organization, and 11 in Instructional Support). 
Items, scored on a scale of 1 to 3 (1 —no, none, never, 2 — few, some, limited, and 3 — most, 
many, varied), correlate with the CLASS dimensions for each tool. 


Most Environment scores averaged in the “few, some, limited” range. The overall infant 
classroom average was 2.15, overall toddler average was 2.19, and overall preschool average 
was 2.46. Table 5 shows the results of the domain and dimension average scores for the CLASS 
Environment assessments for infant, toddler and Pre-K classrooms. 


Table 5. CLASS Environment Average Scores 
cones Environment isle auETERC Score 


PMateriais—SSSSSCSYSCSS BS 
a 


[Materials ——OSSCSCSCS~“~“S*~“‘“~“‘~—rCSS 
[Displays ——SSCSC“‘“‘SCSCSC“‘“*SCSSC‘a SCS 
[Arrangements/Displays/Materals—~—~C~*dtSC“‘(SUN’”CO™C~C*SCOC~‘ aa 
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CLASS Environment Pre-K Classrooms (n=202) | -2a6~—~=~*dtS~=C“‘séODASOC~* 
Displays 
Material 

a 

P Displays SSSCSC~*~“~SC“<~iRSCSC‘“dtSC(“‘C#CSSCSC* 


Displays/Materials 


Coaching 


Program administrators and teachers receive individual feedback, delivered by their assigned 
ECQS. The ECQS meet with the program director for center-based educators to debrief 
assessment scores, review coaching expectations through review of the Coaching Agreement, 
and share how educator goals will be shared. Specialists then engage in coaching with 
educators using the PBC model. The initial educator visit includes discussion of the Coaching 
Agreement and the implementation of different coaching strategies, gathering input into 
preferred coaching strategies to guide coaching activities. Subsequent visits involve sharing 
individual classroom assessment scores, completing individual needs assessments on identified 
teaching practices, and beginning the PBC coaching cycles starting with goal and action 
planning, following with the remain components of focused observation and reflection and 
feedback. 
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Practice-Based Coaching. A 
total of 4,643 coaching visits 
occurred through November 
30, 2023 totaling 2,476.7 hours 
of coaching with classroom 


Figure 1. Practice-Based Coaching Component 
Implementation (n=4,643) 


7 ‘ ie) fo} 
educators. Coaching sessions 34.2% 44.7% 
mostly occurred in-person 
(74.5%). Coaching activities are 
reported for each coachin 
aianensopeal . 8.0% Sz 
interaction with an educator. 13.1% 
ome me} Oriey of coaching = Establishing the Partnership = Focused Observation 
interactions focused on 

Reflection and Feedback = Shared Goals and Action Planning 


Establishing the Partnership 

(44.7%). This follows the expectation for the Practice-Based Coaching model to build 
relationships with educators as the first step of the coaching relationship and to continue that 
throughout all interactions. (See Figure 1.) 


Coaches reported which coaching strategies they used during coaching sessions. Coaches could 
indicate use of multiple strategies during coaching sessions. Most frequently, coaches engaged 
in problem-solving conversations (43.8%), modeling teaching practices (29.2%), video examples 
from myTeachstone (25.4%), and providing other help in the practice setting (25.5%). 


Coaches also self-rated how successful they felt each component of the PBC was implemented 
at each visit. On a scale of 1 (not at all successful) to 5 (very successful), coaches rated overall 
coaching a 4.27 or mostly successful. 


An essential component of Practice-Based Coaching is for programs to self-select their own 
target goals. For infant educators, goals regarding early language support were most common 
(49%), followed by facilitated exploration (25%) and relational climate (14%). Nearly thirty 
percent of coaching visits in infant classrooms focused on teacher talk. For toddler educators, 
most educators chose a goal in the language modeling (28%) or behavior guidance (32%) 
dimensions. In preschool classrooms, educators most frequently set goals in the behavior 
management (25%), concept development (17%), productivity (15%), and language modeling 
(14%) dimensions. 


Goal status was also recorded. Educators created 1,118 goals in 2023. This, combined with the 
visits where goals were modified, comprised 26.8% of coaching visits. Of these, goals were met 
320 times, which was 28.6% of goals set. On-going progress for work on goals constituted the 
remainder of coaching visits. 


Coaching Session Feedback. A Coaching Session Feedback Survey was created to obtain on- 
going feedback from program participants about their experiences in the program. This brief 
survey is designed to be completed at the conclusion of each coaching visit in less than five 
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minutes. Respondents also have the option to request contact by the project coordinator and 
can provide any other feedback to the QAR team. Respondents rated themselves on the 
following questions, ona scale of 1 — No/Not At All to 5 - Extremely: 


To what degree do you feel you were able to decide the goals and steps you are going to 
work on? 

To what degree do you feel you are making progress on your goal? 

The interactions with my coach helped me to improve my practice? 

The discussion with my coach prompted me to reflect about my practice in a meaningful 
way? 


Through November 2023, 2,082 surveys were gathered (44.8%). Overall, program participants 
feel that they are reflecting on their practices (4.49), improving on their practices (4.35), and 
deciding on their own goals (4.37). Educators reported less efficacy in making progress on their 
goals (3.84); however, many of the educators currently served through the project have only 
recently engaged in coaching, so this is not an unexpected finding. 


Comments from educators were gathered on feedback surveys. A sampling of comments 
includes: 


“[My coach] is so pleasant, friendly and helpful. I'm confident she will build strong 
coaching relationships with our teachers.” 

“[My coach] is very approachable and | really enjoy and feel comfortable working with 
her.” 

“[My coach] is very willing to listen to my ideas and help putting them into action.” 
“[My coach] gave me examples of what | did correctly and how | could improve on my 
wording for scaffolding!” 

“Excellent program. This has helped me become a better teacher! | think more teachers 
should be included! Thank you.” 

“[My coach] is an amazing coach. She has made me feel so comfortable and it's easy to 
be myself and actually be in full teacher mode with out(sic) the anxiety of having 
someone watching me.” 

“[My coach] gave me much needed support and advice. She is helping me to be better 
every day and to practice having a better mindset.” 

“| loved all the conversations and reflections | had today during the visit. | feel confident 
going forward with the goals we discussed.” 

“[My coach] has provided me with so much information that | didn't know | needed and 
it has helped me so much in my everyday work life. | have a more meaningful connection 
with my children while helping them grow and | can't thank her enough. I'm very excited 
to see what the future holds and what else | will learn and implement in my classroom.” 
“[My coach] did a great job of explaining our program results with us and helped us. She 
did great on explaining each area and also ways we can work to improve areas of 
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concerns. She did great listening and was very positive. We are excited for her to start 
working with our program!” 

e “Looking forward to taking advantage of all the resources, implementing them in the 
classroom, and sharing what I learned with my co teachers.” 

e “Thank you [coach] for helping me rethink about how the activity went! It was great to 
have another opinion with a different outlook!” 

e “This is so helpful to not only reflect but also to stop and focus on more productive 
planning for my classroom.” 

e “We really appreciate [coach] and how she is so involved and excited to be helping. She 
gives us encouragement and it really helps us want to grow and learn with her! Couldn't 
ask for a better coach 


1” 


Communities of Practice 


In Fall of 2023, the QAR program began Communities of Practice (CoP) for all QAR participants, 
including lead educators, assistants, and administrators. CoPs were designed to create an on- 
line space for participants to engage in meaningful learning, discussion, and collaboration with 
other professionals around a central topic. To determine facilitation topics, ECIC Content 
Specialists reviewed aggregate CLASS data and topics frequently arising in coaching with 
educators. Three CoP interest areas were created: Language and Literacy, Supporting Educator 
Growth, and Classroom Connections. Sessions, conducted in the evenings, were facilitated 
through Canvas, the on-line learning management software through UMKC and participants 
attend via Zoom. CoP sessions are approved for one-hour of clock hour credit through the 
MOPD system. 


Sessions were held in November and December and will continue in 2024. Session topics and 
attendance are reported in Table 6. 
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Table 6. Community of Practice Topic and Attendance. 


Date of Session Topic Audience Attendance 

November 8, 2023 | Supporting Educator Program administrators and 4 
Growth: Pedagogical family childcare providers 
Leadership 

November 7, 2023 | Empathy asa Classroom educators, 14 
Superpower program administrators, and 

family child care providers 
November 9, 2023 | Language and Literacy Classroom educators, 17 


program administrators, and 
family child care providers 


December 5, 2023 | Supporting Educator Program administrators and S 
Growth: Reflective family childcare providers 
Supervision 

December 11, 2023 | Communicating Behavior | Classroom educators, 6 
Expectations So Young program administrators, and 
Children Can Learn family child care providers 

December 18, 2023 | Beyond words and Classroom educators, 11 
conversations program administrators, and 


family child care providers 


Post-session evaluation data from participants were positive overall regarding meeting 
experience and discussion topics. All participants felt that community engagement guidelines 
were followed. When asked how much participants valued the time to get to know one 
another, 75% stated that they did. Most participants (75%) indicated that they identified a new 
strategy or idea to help them in their role. Over half of respondents, 58%, stated that they 
learned about a new resource. 


Comparison Study 


Overview. Program Administration Scale — Second Edition* (PAS) is a research-based 
assessment instrument used to evaluate the leadership and management practices in center- 
based early learning programs. The PAS measures quality on a 7-point scale, with markers 
describing practices from inadequate (1) to excellent (7). The scale is comprised of 25 items 
grouped in 10 subscales: human resources development, personnel cost and allocation, center 
operations, child assessment, fiscal management, program planning and evaluation, family 
partnerships, marketing and public relations, technology, and staff qualifications. To score 
indicators, the assessor interviews the program administrator and reviews documentation to 
verify indicators. 


For the purposes of the QAR Cohort #6 study, some modifications were made with scale 
implementation. First, programs were not required to gather and present documentation at 
assessment. This was to give programs a “soft” introduction to the assessment tool, as well as 
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extend the time programs have to gather documents. Second, the staff qualifications subscale 
was not rated. This allows programs time to focus on leadership and management practices in 
the program. Significant changes to staff qualifications would likely require more time than this 
study offers. 


PAS Results. The PAS was conducted with 22 programs at Time 1 and 14 programs at Time 2. 
Programs were distributed across the state: 8 in the St. Louis region, 5 in central region, 4 in the 
southwest region, 3 in the Kansas City region, and 2 in the northwest region. Assessments were 
completed for Time 1 in April — May 2022 and April — May 2023 for Time 2. 


Initial program overall scores ranged from 2.00 to 5.67, with a mean score of 3.58. National 
PAS data norms” report a mean score of 3.60, indicating that programs in Model 2 align with 
overall national quality. Mean national norm scores are reported in Table 7. 


Table 7. Average PAS Scores at Pre- and Post-Assessment 


Time 1 Time 2 National Norm 
(n=22) (n=14) (n=564) 
Overall PAS Score 3.58 5.03 3.60 
Human Resources Development 2.60 5.48 3.42 
1. Staff Orientation 2.64 6.29 2.88 
2. Supervision and Performance Appraisal 2.45 5.50 3.37 
3. Staff Development 2.73 4.64 4.01 
Personnel Cost and Allocation 735 S33 2-63 
4. Compensation 2.55 3.50 3.04 
5. Benefits 1.73 2.57 2.00 
6. Staffing Patterns and Scheduling 2.73 3.93 2.85 
Center Operations 3.68 5.43 3.30 
7. Facilities Management 4.73 6.00 4.49 
8. Risk Management 4.18 5.86 2.50 
9. Internal Communications 2.14 4.43 2.40 
Child Assessment 3.91 5.07 5.00 
10. Screening for Special Needs 3.73 4.57 4.50 
11. Assessment in Support of Learning 4.09 5.57 5.50 
Fiscal Management 3.98 5.46 3.48 
12. Budget Planning 3.95 5.21 3.35 
13. Accounting Practices 4.00 5.71 3.61 
Program Planning and Evaluation 2.77 5.07 3.22 
14. Program Evaluation 2.95 5.36 3.62 
15. Strategic Planning 2.59 4.79 2.81 
Family Partnerships 4.00 5.04 4.05 
16. Family Communications 3.05 4.79 3.26 
17. Family Support and Involvement 4.95 5.29 4.84 
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Marketing and Public Relations 4.16 4.93 3.78 
18. External Communication 4.36 5.57 3.97 
19. Community Outreach 3.95 4.29 3.58 
Technology 5.84 6.00 4.29 
20. Technological Resources 6.86 6.14 3.68 
21. Use of Technology 4.82 6.56 4.90 


Pre- and Post-Assessment PAS Scores. When comparing matched scores for the 14 programs 
with both pre- and post-assessments, significant gains were seen in all but one subscale. It is 
important to note that the subscale that saw the most statistically significant growth, Human 
Resources Development, was the most frequent focus (46%) of coaching visits. Center 
Operations and Program Planning and Evaluation each represented 10% of coaching visits, 
respectively. Pre- and post-test average scores are reported in Figure 24. 


Figure 2. Pre- and Post- Program Administration Scale Scores (n=14) 


Overall PAS Score*** 
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Center Operations** 
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Coaching Implementation. 
Overall, coaches conducted 144 Figure 3. Practice-Based Coaching Component 


hours of coaching within 161 Implementation - Administrators (n=161) 


administrative coaching visits. 20.4% 
Visits followed the PBC model, 

with time spent in one of more ™~ 

of the four PBC components: 43.6% 

establishing the partnership, 


shared goals and action 18.7% 
planning, focused observation, 


17.2% 

and reflection and feedback. 

Coaches primarily spent time = Establishing the Partnership = Focused Observation 

creating shared goals and action Reflection and Feedback = Shared Goals and Action Planning 
plans at 43.6% of visits. (See 

Figure 3.) 


Comparison of CLASS assessment pre- and post-assessment scores will be reported in 2024. 
Cohort 6 rating cycle will occur in Spring 2024. 


Conclusions 


Examination of the process for developing and launching the QAR program suggests the project 
is meeting objectives in a comprehensive and thorough way. Hiring and onboarding staff has 
become a smooth process with the addition of ECIC staff becoming CLASS Affiliate trainers for 
reliability. The addition of 10 coaches, for a total of 24, has increased capacity for service 
delivery to begin 2024. All staff are reliable on assessment instruments and receive continued 
support and supervision to ensure fidelity to coaching implementation. Rigorous training 
protocols and meaningful, on-going professional development were useful for building 
knowledge, skills, and confidence of the staff. The ECIC QAR leaders were successful in 
navigating the launch of the program towards state-wide implementation. 


The robust baseline data collected thus far in the project are useful for understanding the 
nature of quality early learning programs in Missouri and will inform the further development 
of QAR. Feedback from both coaches and program participants suggest that educators are 
deeply considering aspects of their practice and beginning to make progress on goal 
attainment. PAS data exemplify this, suggesting that targeted coaching interventions can lead 
to successful practice change. 


Assessment and coaching continue to operate at a steady pace. Coaching is monitored to 
maintain fidelity implementation. Beginning in 2024, the proposed model implementation will 
see adjustments to coaching delivery to support state-wide implementation. Continued 
opportunities to build professional knowledge and support of the QAR workforce will be 
developed and continued in 2024, with the continuation of CoP, offering CDA supports for QAR 
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participants, and the implementation of additional Teachstone curricula to support classroom 
practice change. 


Information dissemination is an essential component of the QAR for multiple stakeholders in 
Missouri and across the U.S. Because the OCC and ECIC are choosing to incorporate cutting- 
edge assessment tools and methods, the project is attracting the attention of national thought 
leaders and researchers as an innovative approach. For example, one research opportunity is 
the examination of the relationship between CLASS and CLASS Environment scores and how 
these interactions could impact coaching and program improvement. The ECIC will continue to 
seek opportunities to message the outcomes of the work and valuable findings from the 
process. 


The ECIC is grateful for the opportunity to partner with OCC in developing an early childhood 
quality improvement system for Missouri. The ECIC leadership is committed to providing QAR 
data and grounded insight to inform the process of creating a system that recognizes quality 
programs and practices. Over the coming months, the QAR project team and a group of 
identified stakeholders will support the OCC in designing additional components to the QAR 
project. This development will continue to facilitate the goal of state-wide program 
implementation in fall of 2024. 
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